1. Field of the Invention
The present invention relates generally to the field of automated search and retrieval of text documents. More specifically, the invention relates to thesauri (especially statistical thesauri), to the structures of the statistical thesauri, to methods of forming the statistical thesauri, and to use of the statistical thesauri in query expansion.
2. Related Art
It is known in the field of information retrieval that both precision and recall can be greatly improved when queries are expanded to contain a larger number of good search terms. A thesaurus can be used to increase the number of good search terms.
A statistical thesaurus is a thesaurus which contains terms that are related to the headword by their co-occurrence with the headword in text. This is in contrast to a traditional thesaurus whose terms, synonyms, are related to the headword by meaning.
Recent research has shown that a statistical thesaurus provides good search terms when used for query expansion, while traditional thesauri provide little improvement and may actually hurt overall performance. As an example, FIG. 6 illustrates synonyms for the headword "murder" from a traditional thesaurus, while FIG. 7 illustrates the related concepts from a statistical thesaurus.
Statistical thesauri can also provide related concepts for many terms not found in a traditional thesaurus, including current events. For example, FIG. 8 illustrates the related concepts for the term "Whitewater". This meaning of the term "Whitewater" cannot be found in any traditional thesaurus.
Therefore, a high performance statistical thesaurus is a very useful tool in an information retrieval system. It is to improving the formation, structure and use of statistical thesauri that the present invention is directed.